To answer this question - yes, you should be concerned and I'd recommend to uninstall skype on all devices where you don't need it.

If you are using Outlook's web app, the following filter for uBlock Origin will completely disable Skype's integration and prevent your IP from being resolved:

||*.skype.com^$third-party

How it works

• The call is made using a guest account
• That guest account has blocked you before calling
• You don't need to accept the call in order to have your IP leaked.
• I haven't seen any P2P connection

I believe the reason that it doesn't show up in the call log is because skype isn't expecting a guest account to call someone that they blocked.

It looks like Skype somehow spits out your IP when you receive a call.

Data from Skype's API

Here's some data from Skype's services that happens when you receive a call from them:

Caller information:

[
   {
      "about":null,
      "avatarUrl":null,
      "birthday":null,
      "city":null,
      "country":null,
      "displayname":"skype-resolver.net",
      "emails":[
         
      ],
      "firstname":"skype-resolver.net",
      "gender":"0",
      "homepage":null,
      "jobtitle":null,
      "language":null,
      "lastname":null,
      "mood":null,
      "namespace":"guest",
      "phoneHome":null,
      "phoneMobile":null,
      "phoneOffice":null,
      "province":null,
      "richMood":null,
      "username":"guest:a93b3f62-010a-49ce-817a-b2b1f953aeaa"
   }
]

Conversation information:

{
   "targetLink":"https://azeus1-client-s.gateway.messenger.live.com/v1/users/ME/contacts/8:guest:a93b3f62-010a-49ce-817a-b2b1f953aeaa",
   "id":"8:guest:a93b3f62-010a-49ce-817a-b2b1f953aeaa",
   "type":"Conversation",
   "version":1613596332633,
   "properties":{
      "isemptyconversation":"False",
      "conversationblocked":"True"
   },
   "lastMessage":{
      "from":"https://azeus1-client-s.gateway.messenger.live.com/v1/users/ME/contacts/8:guest:a93b3f62-010a-49ce-817a-b2b1f953aeaa",
      "type":"Message",
      "conversationLink":"https://azeus1-client-s.gateway.messenger.live.com/v1/users/ME/conversations/8:guest:a93b3f62-010a-49ce-817a-b2b1f953aeaa"
   },
   "messages":"https://azeus1-client-s.gateway.messenger.live.com/v1/users/ME/conversations/8:guest:a93b3f62-010a-49ce-817a-b2b1f953aeaa/messages",
   "lastUpdatedMessageId":0,
   "lastUpdatedMessageVersion":0
}

Skype will try to fetch the conversation messages, but since the conversation is blocked, it will return a "400 Bad Request" error with the following JSON:

{"errorCode":201,"message":"conversation blocked"}

Call information:

{
   "participants":{
      "from":{
         "id":"8:guest:a93b3f62-010a-49ce-817a-b2b1f953aeaa",
         "displayName":"skype-resolver.net",
         "endpointId":"removedf-ffff-ffff-ffff-ffffffffffff",
         "languageId":"en-GB",
         "participantId":"8c34d346-59ba-49dc-96f2-295f92b45d31",
         "hidden":false
      },
      "to":{
         "id":"8:[REMOVED]",
         "displayName":null,
         "endpointId":"00000000-0000-0000-0000-000000000000",
         "languageId":null,
         "participantId":"removedf-ffff-ffff-ffff-ffffffffffff",
         "hidden":false
      }
   },
   "callInvitation":{
      "callModalities":[
         "audio"
      ],
      "links":{
         "progress":"https://cc-euwe-08.cc.skype.com:443/cc/v1/incoming/[REMOVED]/24/t/369/progress?i=215",
         "newOffer":"https://cc-euwe-08.cc.skype.com:443/cc/v1/incoming/[REMOVED]/24/t/369/mediaOfferRequest?i=215",
         "mediaAnswer":"https://cc-euwe-08.cc.skype.com:443/cc/v1/incoming/[REMOVED]/24/t/369/mediaAnswer?i=215",
         "acceptance":"https://cc-euwe-08.cc.skype.com:443/cc/v1/incoming/[REMOVED]/24/t/369/accept?i=215",
         "redirection":"https://cc-euwe-08.cc.skype.com:443/cc/v1/incoming/[REMOVED]/24/t/369/redirect?i=215",
         "callController":"http://callcontroller.invalid",
         "callLeg":"https://cc-euwe-08.cc.skype.com:443/cc/v1/incoming/[REMOVED]/24/t/369/reject?i=215",
         "subscribe":"https://broker-euno-07.broker.skype.com/api/v1/subscribe/d19d9eb1-6f54-4980-bdda-4729e82f0a4b/0?i=13",
         "brokerHttpTransport":"http://52.114.77.179/enc"
      },
      "mediaContent":null,
      "replaces":null
   },
   "additionalActionResponses":[
      {
         "url":"https://conv-euwe-10.conv.skype.com:443/conv/[REMOVED - CONVERSATION ID]?i=219&e=[REMOVED]",
         "output":{
            "roster":{
               "participants":{
                  "8:guest:a93b3f62-010a-49ce-817a-b2b1f953aeaa":{
                     "details":{
                        "id":"8:guest:a93b3f62-010a-49ce-817a-b2b1f953aeaa",
                        "displayName":"skype-resolver.net",
                        "endpointId":"00000000-0000-0000-0000-000000000000",
                        "participantId":null,
                        "languageId":null,
                        "hidden":false
                     },
                     "endpoints":{
                        "removedf-ffff-ffff-ffff-ffffffffffff":{
                           "call":{
                              "serverMuteVersion":0
                           },
                           "capabilities":{
                              "cloudAudioVideoConference":"enabled",
                              "cloudScreenSharing":"disabled",
                              "hostlessConference":"disabled",
                              "cloudMerge":"disabled",
                              "additionalModalityOperationLinks":"disabled",
                              "implicitCallback":"disabled",
                              "autoJoinOnConflict":"disabled",
                              "supportsCompressedServicePayload":"disabled",
                              "serverMuteUnmute":"disabled",
                              "supportNgcMediaControl":"disabled"
                           },
                           "participantId":"removedf-ffff-ffff-ffff-ffffffffffff"
                        }
                     },
                     "role":"guest"
                  }
               },
               "type":"MultiPartyEndpoint",
               "sequenceNumber":0,
               "participantCounts":{
                  "totalParticipants":1,
                  "preheatedParticipants":0,
                  "lobbyParticipants":0,
                  "totalPresenters":0,
                  "requestingAttentionPresenters":0,
                  "totalAttendees":0,
                  "requestingAttentionAttendees":0,
                  "overflowAttendeeCount":0
               }
            },
            "conversationController":"https://conv-euwe-10.conv.skype.com:443/conv/[REMOVED - CONVERSATION ID]?i=219&e=[REMOVED]",
            "sequenceNumber":1,
            "subject":"",
            "activeModalities":{
               "call":null
            },
            "state":{
               "isMultiParty":false,
               "groupCallInitiator":null,
               "isBroadcast":false,
               "isVoiceDataCollectionOn":false
            },
            "links":{
               "leave":"https://conv-euwe-10.conv.skype.com:443/conv/[REMOVED - CONVERSATION ID]/leave?i=219&e=[REMOVED]",
               "addParticipant":"https://conv-euwe-10.conv.skype.com:443/conv/[REMOVED - CONVERSATION ID]/addParticipant?i=219&e=[REMOVED]",
               "removeParticipant":"https://conv-euwe-10.conv.skype.com:443/conv/[REMOVED - CONVERSATION ID]/removeParticipant?i=219&e=[REMOVED]",
               "addModality":"https://conv-euwe-10.conv.skype.com:443/conv/[REMOVED - CONVERSATION ID]/addModality?i=219&e=[REMOVED]",
               "addParticipantAndModality":"https://conv-euwe-10.conv.skype.com:443/conv/[REMOVED - CONVERSATION ID]/add?i=219&e=[REMOVED]",
               "removeModality":"https://conv-euwe-10.conv.skype.com:443/conv/[REMOVED - CONVERSATION ID]/removeModality?i=219&e=[REMOVED]",
               "mute":"https://conv-euwe-10.conv.skype.com:443/conv/[REMOVED - CONVERSATION ID]/mute?i=219&e=[REMOVED]",
               "unmute":"https://conv-euwe-10.conv.skype.com:443/conv/[REMOVED - CONVERSATION ID]/unmute?i=219&e=[REMOVED]",
               "notificationLinks":"https://conv-euwe-10.conv.skype.com:443/conv/[REMOVED - CONVERSATION ID]/notificationLinks?i=219&e=[REMOVED]",
               "merge":"https://conv-euwe-10.conv.skype.com:443/conv/[REMOVED - CONVERSATION ID]/merge?i=219&e=[REMOVED]",
               "updateEndpointMetadata":"https://conv-euwe-10.conv.skype.com:443/conv/[REMOVED - CONVERSATION ID]/updateEndpointMetadata?i=219&e=[REMOVED]",
               "updateEndpointState":"https://conv-euwe-10.conv.skype.com:443/conv/[REMOVED - CONVERSATION ID]/updateEndpointState?i=219&e=[REMOVED]",
               "admit":"https://conv-euwe-10.conv.skype.com:443/conv/[REMOVED - CONVERSATION ID]/admit?i=219&e=[REMOVED]",
               "conversationHttpTransport":"http://52.114.74.144/enc",
               "publishState":"https://conv-euwe-10.conv.skype.com:443/conv/[REMOVED - CONVERSATION ID]/publishState?i=219&e=[REMOVED]",
               "removeState":"https://conv-euwe-10.conv.skype.com:443/conv/[REMOVED - CONVERSATION ID]/removeState?i=219&e=[REMOVED]",
               "updateMeetingSettings":"https://conv-euwe-10.conv.skype.com:443/conv/[REMOVED - CONVERSATION ID]/updateMeetingSettings?i=219&e=[REMOVED]",
               "searchParticipants":"https://conv-euwe-10.conv.skype.com:443/conv/[REMOVED - CONVERSATION ID]/searchParticipants?i=219&e=[REMOVED]",
               "getAllParticipants":"https://conv-euwe-10.conv.skype.com:443/conv/[REMOVED - CONVERSATION ID]/getAllParticipants?i=219&e=[REMOVED]",
               "admitAll":"https://conv-euwe-10.conv.skype.com:443/conv/[REMOVED - CONVERSATION ID]/admitAll?i=219&e=[REMOVED]"
            },
            "capabilities":{
               "cloudAudioVideoConference":"enabled",
               "cloudScreenSharing":"disabled",
               "hostlessConference":"disabled",
               "cloudMerge":"disabled",
               "additionalModalityOperationLinks":"disabled",
               "implicitCallback":"disabled",
               "autoJoinOnConflict":"disabled",
               "supportsCompressedServicePayload":"disabled",
               "serverMuteUnmute":"disabled",
               "supportNgcMediaControl":"disabled"
            },
            "subscriptionDetails":{
               "selfParticipant":{
                  "version":0,
                  "state":"active",
                  "details":{
                     "id":"8:[REMOVED]",
                     "displayName":"[REMOVED]",
                     "endpointId":"00000000-0000-0000-0000-000000000000",
                     "participantId":null,
                     "languageId":null,
                     "hidden":false
                  },
                  "endpoints":null,
                  "role":"admin"
               }
            }
         }
      }
   ],
   "debugContent":{
      "callId":"removedf-ffff-ffff-ffff-ffffffffffff",
      "participantId":"removedf-ffff-ffff-ffff-ffffffffffff"
   }
}

Outlook's web app also appears to reject the call even though it doesn't seem to reject it with the following request:

{callEnd: {code: 410, subCode: 3111, phrase: "CallEndReasonMediaOfferProcessingError"}}