In theory - yes, your computer can still be hacked even, if all TCP and UDP ports are closed and you don't have any malware running on it. However, odds of getting your computer hacked, if all ports are closed, is way lower compared to a system where a web server is listening on publicly accessible port.
To give an example, there are some protocols that don't have concept of ports in the first place, for example, ICMP, IPv4 or Ethernet. If your Operating System's Networking Stack has a buggy ICMP, IPv4 or Ethernet implementation, then, in theory, such bug might be used as backdoors to gain partial or full control of your computer. See Ping of Death bug that gave attackers control to crash arbitrary computers with vulnerable ICMP implementations even if no UDP or TCP ports were open.
Overall, if a computer can be hacked, then it always is:
- an implementation bug (e.g. software engineer made a mistake in his code. See OpenSSL Heartbleed vulnerability or Ping of Death vulnerability);
- a design bug (e.g. people designing the protocol did not think carefully enough about authentication aspects; See SSLv3 POODLE attack or WEP protocol vulnerabilities); or
- a bug in instructions that administrator uses to provision the host (e.g. using weak passwords; or running certain software that does not enforce authentication in untrusted zone; or simply [un]intentionally running old and buggy software)
When evaluating security, I always recommend other software engineers to think in terms of
How much execution flow of the code running on system attacker can control, before your code
will drop this IP packet as malicious. The earlier this drop of malicious packet happens the more secure the system is.
For example, if all ports are closed, then attacker can still "execute" kernel-space code up to TCP and UDP protocol handling where packet would eventually be dropped because there is no open destination port - while very unlikely, there still could be bugs in the IP or TCP handling code in the kernel that no one today is aware of. However, if you have open ports, then this means that attacker could try to leverage bugs not only in the kernel space but also in the user space (ie HTTP implementation - the Web Server listening on port 80)
Also, the other answer mentions potential security bugs in the "client" side code, for example, Web Browser. These are harder to leverage because attacker would need to be in the path between your host and the server it tries to talk to; then intercept HTTP session; and alter communication in hopes to expose a potential bug in your web browser. However, in you question you have premise "knowing only IP address" which I interpret as "Another guy from different physical location trying to hack your computer". However, if this other guy would be in the same L2 broadcast domain then he can spoof ARP to suddenly be in the path between your browser and server; Effectively he could leverage a new set of bugs that exist in your browser.