This is a major revision to my previous answer, which was factually incorrect. It is posted as a new answer in order to prevent baiting-and-switching of votes, as I feel the current answer is not the same as the one I had initially posted.
In order for a social media application to notify a user that a screenshot has been taken, the application has to know that a screenshot is being taken.
To my knowledge, neither iOS, nor Android, have a framework intended to detect screenshots. This means that developers need to find a workaround on how to detect a screenshot being taken. There are several questions on Stack Exchange regarding this (such as here and here) and the method that seems to be recommended the most is to observe for a new file to be created in the screenshots directory.
Since an application can determine if it is currently in foreground or not, and can receive a notification when a new screenshot is being created, the application can therefore determine if a screenshot of the application is being taken. The application can then react accordingly, such as notifying users that a screenshot of their profile or conversation has been taken.
Keep in mind that I cannot answer exactly if this is how Instagram or other Social Media applications implement this. This answer explains how it could be done.