You seem to be confused about the relationship between refresh tokens and JWTs (which are frequently used as access tokens), which is reflected in the fundamental differences between how refresh and access tokens are used. I'll explain that below, and the security considerations of the relationship and uses, in a way that will hopefully resolve your confusion.
The term "access token" frequently refers to a JWT (JSON Web Token). In OAuth this is practically mandatory, though JWTs can be (and frequently are) used outside of OAuth as well. JWTs are stateless; the server has no idea what JWTs exist at any point in time, who they were issued to, when they'll expire, or what access they authorize. Once a JWT is issued, the server forgets it entirely (beyond hopefully noting the event in an audit log).
Stateless tokens have a big advantage in scalability: the server neither needs to store state for each user within itself (which runs into problems with large user counts and also makes load-balancing and failover more difficult) nor retrieve session data from an external data source like a database (which increases request/response latency and load on the DB server). Instead, the token can store everything that the server needs to know about the user, with no lookups required. However, stateless tokens also have a huge disadvantage in security: there is no good way to terminate them ahead of their scheduled expiration, because the server has no idea which tokens are valid. This is a problem for users (the user can't revoke access to potentially-stolen tokens) and for administrators (the admin can't end a logged-in user's access prematurely if e.g. they leave the company or break some rule and need to be banned... or if their token is suspected to be stolen).
The usual fix to that problem is to make the stateless tokens very short-lived. Specifically, a matter of minutes is typical (a week is absolutely FAR too long; even an hour is pushing it). The idea being that any given stateless token will expire so quickly, that even if it is stolen, misused, or granted to somebody whose access must be revoked, the window of opportunity for harm is very short. However, this would be a terrible user experience if it meant you needed to log in (or even bounce off an authorization server / SSO provider) every few minutes. To solve this problem, we introduce refresh tokens.
Refresh tokens are, in a sense, a return to the classic session token. Refresh tokens are generally opaque high-entropy blobs; their contents mean nothing, but can be looked up in a database somewhere. Because the refresh token needs to be stored in the backend (typically in a DB), it's not stateless. However, since it's only needed once every few minutes per active user, it adds minimal load to the DB server and basically no delay (none at all for continuously-active sessions, if they're built right) to normal user actions. One needn't even transmit the refresh token except when receiving it from the server, or the client exchanges it for a new stateless token (i.e. access token). The access token is used for all other requests. Meanwhile, the refresh token can sit on the client and the database for as long as you want a single authentication to last - which could potentially be years (I think StackExchange uses 6 months since last activity). If either the user or an admin want to change or revoke existing sessions, they do this by removing the refresh token(s) or modifying their associated account. Within minutes the stateless access token(s) will expire, and the client will either be refreshed with a new access token that has different privileges, or with the notice that their refresh token is invalid and they need to sign in again (or their account is locked, or whatever).
Hopefully this helps you understand the differences in best practice regarding lifetimes of the two token types. Refresh tokens are designed to be long-lived but must be revoked at need. Access tokens are designed to be short-lived, because they can't be revoked (in most cases). As for the security usefulness of each token, it's a little complicated. Access tokens are more valuable in the very short term - they can't be revoked easily, directly provide authorization, and don't even require checking an auth server - but they expire so fast they're difficult to mount an effective attack around. Refresh tokens are much better for persistent access... assuming nobody notices and revokes them, which is quite easy.