It won’t be pretty but you’re stuck with what you have done. Best choice is probably to use the horizontal (landscape) footage as is and add a neutral background for the vertical (portrait) shots. Anything you do to crop or “expand to fill frame” will result in loosing parts of the image.
Rather than a neutral background, some editors are blowing up the vertical footage and using this as a background, with the original portrait footage overlaid in front. For my taste this is a bit too busy; I prefer the neutral.
