There’s a risk but not high. 

You may have noticed pre-cum on an erection. This fluid reduces the acidity in the urethra increasing the chances of survival for sperm during an ejaculation. Before cumming, all penises leak this fluid. 

While this fluid itself doesn’t contain any sperm, it may pick up any sperm through recent ejaculations, and this sperm can cause pregnancy. Visit your doctor just to be on the safer side.

By the way, did you give him your consent to take his condom off? If no, then you should know it is a serious issue! Taking a condom during sex without you finding out is called stealthing and it amounts to having sex without consent. 

You gave your consent to have sex if he wears a condom during the entire session, not without one. Also, since you met this person for the first time, get yourself tested for STIs/STDs just to be on the safer side.