Yes, but don't do it.
If you are asking if it is possible to come up with a proprietary scheme for encrypting data that only your application "knows," the answer is yes, until someone figures out how to reverse-engineer it (which is usually easier than you realize). This sort of practice is known as security by obscurity and is highly discouraged.
Instead, your application should use an encryption scheme that is public domain (and therefore proven and well-tested). To provide application-specifity, use a cryptographic key that only the application "knows." A hacker attempting to reverse engineer the key may have some success, but the paramters governing that success (e.g. how long it will take and the computing power required) are well known and understood and you can devise an overall risk strategy appropriate to the level of security required by your business case (e.g. you can use longer keys or change them more often).
Encryption is only part of the problem
Not sure what attack vectors you are attempting to mitigate, but I'm guessing you want an "image viewer" application that is the only means to viewing an image, and in addition you are working with the assumption that the application can fall into a hacker's hands. If these are correct, you are up against a very difficult problem (compromised client endpoint).
Usually if there is a compromised client endpoint, us security professionals give up. A fully bullet-proof solution is nearly impossible. But attempts have been made, and these fall under the lexicon of Digital Rights Management (DRM). Here's a few of the problems you will need to solve:
- You need a protected media path.
Even if all the security in your app works, it still needs to display the image, and the way your O/S displays an image is using a video driver. Well, anyone can write their own video driver, and there is nothing stopping a video driver from recording and storing an image for unintended use. The typical way to deal with this is to require the video driver libraries to be digitally signed, and for the O/S to verify the signature before allowing output to go to the driver.
- You need to protect your application's memory
No matter how clever your app is, a hacker can find a way to get access to its memory if he has physical access to the machine. A bare minimum mitigation for this is to run a little code to check to see if any debuggers are currently running on the system, and halt output if there are. This is not likely to stop a determined hacker who has ways to mask such things.
- You need to protect the content and the key
Solutions I have seen involve storing the media in encrypted format (using a symmetric key-- a public/private key is way too slow). When the viewer runs, it requests a key from a server, using a signed document indicating that it has a right to request the key. The key is stored in memory and the content is decrypted, then the key is overwritten in memory with random data and discarded.
- You need to stop the hacker from taking a photograph of the video monitor
Pretty hard problem to solve. Even if you do all this security stuff, nothing in the world will stop someone from pulling out their camera and taking a shot of the screen. This would be lossy, of course, but with a good monitor and good camera you can still get a great image.
The only mitigation for this I can think of is to include a contrast-sensitive watermark on the image itself. This may or may not be acceptable to you.