If your ex has visitation based upon an order of the Court, you can be held in contempt of that order for withholding the children. If there is no order, but just an agreement between the parties, there would be no contempt.
If there is an order, you can file a motion to amend the visitation and ask for an expedited hearing to bring your concerns to the attention of the court. You could ask the court to temporarily (pending a hearing) suspend visitation, order no consumption of alcohol while the children are in his care, etc.
If there is no order, you can file a motion to establish a visitation order, requesting certain stipulations to address your concerns
f you are ever concerned about the safety or wellbeing of the children while in his custody, you can always contact your local law enforcement agency to conduct a welfare check on the children.
The above response has been provided for general guidance on matters of interest only and is not intended as a legal advice. No attorney-client relationship has been created between the reader of this response and The Law Office of Karen M. Holman, PLLC.