I would say that it is closer to “très rarement”, because of the presence of “Je crains de”. For example “Je n’ai pas souvent quitté mon pays” would be close to “de temps en temps”.

In both cases it depends on the context, especially on the person saying that in regard of his language habits.