Cross-posted from MSE where the question didn't get much attention. The question is related to and has a similar motivation as this MO question.
Let$\newcommand{\from}{\colon}\newcommand{\sgn}{\mathrm{sgn}}\newcommand{\defby}{:=}\newcommand{\after}{\circ}\newcommand{\Vec}{\mathbf{Vec}}\newcommand{\Hom}{\mathrm{Hom}}\newcommand{\L}{\mathbf{L}}\newcommand{\Set}{\mathbf{Set}}$ $X$ be a finite set and let $\sigma \from X \to X$ be a map. When $\sigma$ is bijective, the definition of $\sgn(\sigma) \in \{\pm1\}$ is well-known, I propose to define furthermore $\sgn(\sigma) \defby 0$ if $\sigma$ is not bijective. Then the law $\sgn(\sigma \after \tau) = \sgn(\sigma)\sgn(\tau)$ remains true. Also, we still have $\sgn(\sigma) = \det(L_\sigma)$ where $L_\sigma \from \mathbb{R}^X \to \mathbb{R}^ X$ is the induced map given by acting on the basis vectors according to $\sigma$.
Speaking of determinants, in linear algebra you can do a little bit more than defining determinants of an endomorphism: You have a functor $\det \from \Vec_n \to \Vec_1$ from the category of $n$-dimensional vector spaces to the category of $1$-dimensional vector spaces which associates to every $n$-dimensional vector space $V$ its determinant line $\det(V) \defby \bigwedge^n V$ and to a morphism $f \from V \to W$ of $n$-dimensional vector spaces the map $\det(f) \defby \bigwedge^n f \from \det(V) \to \det(W)$. If $V = W$ then the composition $\Hom(V, V) \to \Hom(\det(V), \det(V)) \to \mathbb{R}$ recovers the usual definition of the determinant of an endomorphism. Here the isomorphism $\Hom(L, L) \to \mathbb{R}$ for a line $L$ is given by $f \mapsto f(e) / e$ where $\{e\}$ is some basis for $L$, the choice doesn't matter.
Given an arbitrary morphism $f \from V \to W$ in $\Vec_n$ it is still true that $f$ is an isomorphism if and only if $\det(f)$ is an isomorphism. Moreover, even though $\det(f)$ is not a number, given two morphisms $f$ and $g$ (and, say, $g$ an isomorphism) we can still consider the quotient $\det(f) / \det(g) \in \mathbb{R}$ and determine if $f$ and $g$ have "the same orientation" or more generally, if they increase the signed volume of some parallelepiped by the same amount.
I wonder if the same also works for sets: Is there some category $\L$ (the category $\Set_1$ of one-element sets won't work) together with a functor $\det \from \Set_n \to \L$ such that it makes sense to speak about the determinant of an arbitrary map of sets $X \to Y$ of cardinality $n$ such that we can recover the sign in the case of an endomorphism. In fact, the answer is yes by 2. below, but I would like to have a more basic combinatorial description of this functor.
Of course, it is enough for $\L$ to consist of a single isomorphism class. Moreover, the unique object $L$ (up to isomorphism) of $\L$ should have the property that $\Hom(L, L) \cong \{-1, 0, 1\}$.
Some remarks:
If we were to restrict our attention to the core $(\mathbf{Set}_n)_{\cong}$ (having only bijections as morphisms), then we could choose $\L$ to be the core of $\Set_2$: Objects are two-element sets and morphisms are bijections. We have $\Hom(L, L) \cong \{\pm1\}$ for an object $L$ of this category. A construction of a reasonable functor $\det \from (\Set_n)_{\cong} \to (\Set_2)_{\cong}$ is indicated e.g. in this MO answer: One can define $\det(X) \defby T_X / {\sim}$ with $T_X$ and $\sim$ defined as in the linked answer. However, I don't see how to make this work also for non-isomorphisms.
A definition which does work is $\L \defby \Vec(\mathbb{F}_3)_1$, the category of $1$-dimensional $\mathbb{F}_3$-vector spaces. We have $\Hom(L, L) \cong \{-1, 0, 1\}$ in this category as required. Furthermore, we can use the functor $X \mapsto \det(\mathbb{F}_3^X)$, the determinant of an $\mathbb{F}_3$-vector space being defined simiarly as in the real case above. What I don't like here is that there is so much linear algebra involved for a basically set-theoretic question. Moreover the entrance of the field $\mathbb{F}_3$ seems very ad-hoc.
An equivalent and more elementary way to describe the category $\L$ in 2. is as the category of free-and-transitive $M$-sets for the monoid $M = \{-1, 0, 1\}$. What would be nice then, would be a linear-algebra-free construction of the $\det$ functor.
I've read that $\mathbb{F}_1$-geometry might be relevant to such questions, but unfortunately, I know nothing about it.