One major difference between Access Tokens and Refresh Tokens is who gets to see them.
The Access Token is handed out by the Authorization Server, and is then used by the client to gain access to the Resource Owner's resoures. The Access Token is seen by both the Authorization Server, the Client and the Resource Server.
The Refresh Token is handed out by the Authorization Server, and is only seen by the Client and the Authorization Server. The Resource Server never sees the Refresh Token.
So, the Access Token is more vulnerable; it is seen by more parties. A sloppy implementation on the Resource Server could lead to the leaking of the Access Token, while the Refresh Token would still be safe.
Credit: I had some help from this Stack Overflow answer writing this.
One scenario it mentions is that the Access Token might be sent as a query parameter, and then end up in a log file. The Access Token was protected by HTTPS during transit, but is now in the log file on the Resource Server. People who are not entitled to knowing the Access Token, might be entitled to see the log file. For example, students having a summer job as tech support.
What's important, though, isn't to try and invent a specific scenario in which things might go wrong. What's important is to realize that the Access token could leak with the Refresh Token still being safe. It's a form of defensive programming - be prepared for the case that there is a vulnerability in the Resource Server.
Since in the given situation the Authorization Server and Resource Server are a single application, most scenarios where the Access Token is compromised will also have a compromise of the Refresh Token.
As a side note, I've worked on a similar scenario. We used Identity Server as the Authorization Server. It was built into the application. If that application leaked, I would look to the Resource Server part first. Identity Server is built by security experts and, being Open Source, has been scrutinized by many. The Resource Server part of that particular application was proprietary. You may want to look at Identity Server for your application as well.