An instance reset is a way of fully or partially restarting an instance. This way, mobs will come back or, in some cases, unbug.
Soft reset[]
A soft reset is when all party members leave the instance, and then come back in. This puts people out of combat, making mobs run back to their original places.
The term "soft reset" can also mean having all party/raid members leave the instance for at least 30 minutes. This causes any trash mobs that have been killed to respawn. The most common reason to do this (deliberately) is to reset a boss fight that bugged out part way through the encounter.
Hard reset[]
A hard reset is when all party members leave the instance, and the instance leader right clicks over his own portrait and the clicks the "Reset Instance" button. This restarts the whole instance, and all mobs will be back.
Raid instances cannot be hard reset manually. All heroic mode instances (and thus, the IDs) will hard reset at 7:59 AM Pacific time every day. The date of the hard reset of raid instances is always Tuesday though the actual time depends on what Blizzard is doing (i.e. standard restart, deploying new content patches, or standard maintenance). Unless a player has manually chosen to lock out the raid, in which case the raid ID will be saved even through Tuesday restarts.
If you are in a heroic instance at the time of the reset you will be forcibly returned to your hearthstone's home. At 60, 45 and 15 minutes before the reset there is a warning that the instance will reset. At 10 minutes before the reset you will be informed that you will be forcibly returned to your hearthstone's home.
When resetting a normal instance, some restrictions apply. You cannot reset an instance more than five times per hour. This was added to prevent farming an instance's trash mobs over and over again without killing a boss. This is by 11/05/14 apparantly not the case anymore. Vortex Pinnacle is now on its 15th reset within the hour.
Resetting in Heroic Instances will reset trash but not bosses.