If you're using Outlook as your native email, calendar, or contacts app, you're in for a treat. Indeed, a recent update to the application has introduced the ability to delete contacts from any of your accounts added to it (via Thurrott.com.)
It is still quite difficult to find this setting, as Microsoft replaced the direct Contacts tab in the Outlook Android app with the new search experience. You'll first need to head to the search tab, and then click the tiny "Contacts" link at the top left side of your screen, directly under the search bar. From there you can find a contact, edit its details and now delete it.
Interestingly, this feature was not documented in the version 2.2.181 changelog for Outlook in the Google Play Store. Microsoft instead made note of it on a separate post on the Office Support page that documents what's new in Office 365.