Should fieldset element be used only in form?

Briefly should not be used outside of a form, for the sake of semantics! It is assumed that fieldset is a set of elements that make up part or a whole of a form. Using it outside of a form breaks the semantics suggested by W3C

Many assistive technologies will use the <legend> element as if it were part of each widget's label within the corresponding <fieldset> element. For example, some screen readers, such as Jaws or NVDA , they will pronounce the contents of the caption before pronouncing the label of each widget.

The fieldset needs an element legend inside. This may or may not be a problem for vc, but for screen readers the absence of it is sure to be an inconvenience...

source: https://developer.mozilla.org/pt-BR/docs/Web/Guide/HTML/Forms/How_to_structure_an_HTML_form

Although even using the fieldset outside of a form there are no problems of validation of HTML is not indicated, since inside it there should be inputs, and other form elements. https://validator.w3.org/nu/#textarea

OBS:

If you want to group elements that are not form vc you can use ul/li, or tables, or dl/dd, or section/article and so on...

There is what works and what is right. It makes no mistake to use outside of a form, but it makes no sense to use outside.

This tag exists to give a better semantics to what is being done, even meets the accessibility requirements. There are other ways to group elements, but this one says that these elements are a group of fields and this is important for your SEO and make it more accessible.

Use to determine something that refers to all fields in the group. In addition to styling that could be done with another tag can do specific actions such as disabling all fields in this group.

If you want to do it right just use inside a <form>.

Documentation .

Example that works and is not in form (do not do this at home):

fieldset {
  background-color: #D0D0D0;
  max-width: 300px;
  padding: 12px;	
}

<div class>Título</div>
<fieldset>
Não usei em formulário
</fieldset> 

Correct use:

fieldset {
  background-color: #D0D0D0;
  max-width: 300px;
  padding: 12px;	
}

<form>
   <fieldset>
      <legend>Endereço</legend>
      Tipo Logradouro: <input type = "text"><br/>
      Logradouro:<input type = "text"><br/>
      Número:<input type = "text">
   </fieldset>
</form>

I put on GitHub for future reference.